Causal and Associational Language in Observational Health Research: A Systematic Evaluation.

We estimated the degree to which language used in the high-profile medical/public health/epidemiology literature implied causality using language linking exposures to outcomes and action recommendations; examined disconnects between language and recommendations; identified the most common linking phrases; and estimated how strongly linking phrases imply causality. We searched for and screened 1,170 articles from 18 high-profile journals (65 per journal) published from 2010-2019. Based on written framing and systematic guidance, 3 reviewers rated the degree of causality implied in abstracts and full text for exposure/outcome linking language and action recommendations. Reviewers rated the causal implication of exposure/outcome linking language as none (no causal implication) in 13.8%, weak in 34.2%, moderate in 33.2%, and strong in 18.7% of abstracts. The implied causality of action recommendations was higher than the implied causality of linking sentences for 44.5% or commensurate for 40.3% of articles. The most common linking word in abstracts was "associate" (45.7%). Reviewers' ratings of linking word roots were highly heterogeneous; over half of reviewers rated "association" as having at least some causal implication. This research undercuts the assumption that avoiding "causal" words leads to clarity of interpretation in medical research.

DAG With Omitted Objects Displayed (DAGWOOD): a framework for revealing causal assumptions in DAGs.

Directed acyclic graphs (DAGs) are frequently used in epidemiology as a method to encode causal inference assumptions. We propose the DAGWOOD framework to bring many of those encoded assumptions to the forefront. DAGWOOD combines a root DAG (the DAG in the proposed analysis) and a set of branch DAGs (alternative hidden assumptions to the root DAG). All branch DAGs share a common ruleset, and must 1) change the root DAG, 2) be a valid DAG, and either 3a) change the minimally sufficient adjustment set or 3b) change the number of frontdoor paths. Branch DAGs comprise a list of assumptions which must be justified as negligible. We define two types of branch DAGs: exclusion branch DAGs add a single- or bidirectional pathway between two nodes in the root DAG (e.g., direct pathways and colliders), while misdirection branch DAGs represent alternative pathways that could be drawn between objects (e.g., creating a collider by reversing the direction of causation for a controlled confounder). The DAGWOOD framework 1) organizes causal model assumptions, 2) reinforces best DAG practices, 3) provides a framework for evaluation of causal models, and 4) can be used for generating causal models.

Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength.

INTRODUCTION: Assessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment. METHODS: We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on 26 November 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation. RESULTS: After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-sectional), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes. DISCUSSION: The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigour to be actionable by policy-makers. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

Association between convalescent plasma treatment and mortality in COVID-19: a collaborative systematic review and meta-analysis of randomized clinical trials.

BACKGROUND: Convalescent plasma has been widely used to treat COVID-19 and is under investigation in numerous randomized clinical trials, but results are publicly available only for a small number of trials. The objective of this study was to assess the benefits of convalescent plasma treatment compared to placebo or no treatment and all-cause mortality in patients with COVID-19, using data from all available randomized clinical trials, including unpublished and ongoing trials (Open Science Framework, https://doi.org/10.17605/OSF.IO/GEHFX ). METHODS: In this collaborative systematic review and meta-analysis, clinical trial registries (ClinicalTrials.gov, WHO International Clinical Trials Registry Platform), the Cochrane COVID-19 register, the LOVE database, and PubMed were searched until April 8, 2021. Investigators of trials registered by March 1, 2021, without published results were contacted via email. Eligible were ongoing, discontinued and completed randomized clinical trials that compared convalescent plasma with placebo or no treatment in COVID-19 patients, regardless of setting or treatment schedule. Aggregated mortality data were extracted from publications or provided by investigators of unpublished trials and combined using the Hartung-Knapp-Sidik-Jonkman random effects model. We investigated the contribution of unpublished trials to the overall evidence. RESULTS: A total of 16,477 patients were included in 33 trials (20 unpublished with 3190 patients, 13 published with 13,287 patients). 32 trials enrolled only hospitalized patients (including 3 with only intensive care unit patients). Risk of bias was low for 29/33 trials. Of 8495 patients who received convalescent plasma, 1997 died (23%), and of 7982 control patients, 1952 died (24%). The combined risk ratio for all-cause mortality was 0.97 (95% confidence interval: 0.92; 1.02) with between-study heterogeneity not beyond chance (I2 = 0%). The RECOVERY trial had 69.8% and the unpublished evidence 25.3% of the weight in the meta-analysis. CONCLUSIONS: Convalescent plasma treatment of patients with COVID-19 did not reduce all-cause mortality. These results provide strong evidence that convalescent plasma treatment for patients with COVID-19 should not be used outside of randomized trials. Evidence synthesis from collaborations among trial investigators can inform both evidence generation and evidence application in patient care.

Much ado about something: a response to "COVID-19: underpowered randomised trials, or no randomised trials?"

Non-pharmaceutical interventions (NPI) for infectious diseases such as COVID-19 are particularly challenging given the complexities of what is both practical and ethical to randomize. We are often faced with the difficult decision between having weak trials or not having a trial at all. In a recent article, Dr. Atle Fretheim argues that statistically underpowered studies are still valuable, particularly in conjunction with other similar studies in meta-analysis in the context of the DANMASK-19 trial, asking "Surely, some trial evidence must be better than no trial evidence?" However, informative trials are not always feasible, and feasible trials are not always informative. In some cases, even a well-conducted but weakly designed and/or underpowered trial such as DANMASK-19 may be uninformative or worse, both individually and in a body of literature. Meta-analysis, for example, can only resolve issues of statistical power if there is a reasonable expectation of compatible well-designed trials. Uninformative designs may also invite misinformation. Here, we make the case that-when considering informativeness, ethics, and opportunity costs in addition to statistical power-"nothing" is often the better choice.

Improvements in transition times through the HIV cascade of care among gay and bisexual men with a new HIV diagnosis in New South Wales and Victoria, Australia (2012-19): a longitudinal cohort study.

BACKGROUND: Most studies assessing the HIV care cascade have typically been cross-sectional analyses, which do not capture the transition time to subsequent stages. We aimed to assess the longitudinal HIV cascade of care in Australia, and changes over time in transition times and associated factors. METHODS: In this longitudinal cohort study, we included linked data for gay and bisexual men (GBM) with a new HIV diagnosis who attended clinics participating in the Australian Collaboration for Coordinated Enhanced Sentinel Surveillance in New South Wales and Victoria between Jan 1, 2012, and Dec 31, 2019. We assessed three cascade transition periods: diagnosis to linkage to care (stage 1 transition); linkage to care to antiretroviral therapy (ART) initiation (stage 2 transition); and ART initiation to virological suppression (viral load ≤200 copies per mL; stage 3 transition). We also calculated the probability of remaining virologically suppressed after the first recorded viral load of less than 200 copies per mL. We used the Kaplan-Meier method to estimate transition times and cumulative probability of stage transition. FINDINGS: We included 2196 GBM newly diagnosed with HIV between 2012 and 2019 contributing 6747 person-years of follow-up in our analysis. Median time from HIV diagnosis to linkage to care (stage 1 transition) was 2 days (IQR 1-3). Median time from linkage to care to ART initiation (stage 2 transition) was 33 days (30-35). Median time from ART initiation to first recorded virological suppression (stage 3 transition) was 49 days (47-52). The cumulative probability of ART initiation within 90 days of linkage to care increased from 36·9% (95% CI 32·9-40·6) in the 2012-13 calendar period to 94·1% (91·2-96·0) in the 2018-19 calendar period and cumulative probability of virological suppression within 90 days of ART initiation increased from 54·3% (48·8-59·3) in the 2012-13 calendar period to 82·9% (78·4-86·4) in the 2018-19 calendar period. 91·6% (90·1-93·1) of GBM remained virologically supressed up to 2 years after their first recorded virological suppression event. INTERPRETATION: In countries with high cross-sectional cascade estimates such as Australia, the impact of treatment as prevention is better estimated using longitudinal cascade analyses. FUNDING: National Health and Medical Research Council Australia.

Impact Evaluation of Coronavirus Disease 2019 Policy: A Guide to Common Design Issues.

Policy responses to coronavirus disease 2019 (COVID-19), particularly those related to nonpharmaceutical interventions, are unprecedented in scale and scope. However, evaluations of policy impacts require a complex combination of circumstance, study design, data, statistics, and analysis. Beyond the issues that are faced for any policy, evaluation of COVID-19 policies is complicated by additional challenges related to infectious disease dynamics and a multiplicity of interventions. The methods needed for policy-level impact evaluation are not often used or taught in epidemiology, and they differ in important ways that may not be obvious. Methodological complications of policy evaluations can make it difficult for decision-makers and researchers to synthesize and evaluate the strength of the evidence in COVID-19 health policy papers. Here we 1) introduce the basic suite of policy-impact evaluation designs for observational data, including cross-sectional analyses, pre-/post- analyses, interrupted time-series analysis, and difference-in-differences analysis; 2) demonstrate key ways in which the requirements and assumptions underlying these designs are often violated in the context of COVID-19; and 3) provide decision-makers and reviewers with a conceptual and graphical guide to identifying these key violations. Our overall goal is to help epidemiologists, policy-makers, journal editors, journalists, researchers, and other research consumers understand and weigh the strengths and limitations of evidence.

Association of Convalescent Plasma Treatment With Clinical Outcomes in Patients With COVID-19: A Systematic Review and Meta-analysis.

Importance: Convalescent plasma is a proposed treatment for COVID-19. Objective: To assess clinical outcomes with convalescent plasma treatment vs placebo or standard of care in peer-reviewed and preprint publications or press releases of randomized clinical trials (RCTs). Data Sources: PubMed, the Cochrane COVID-19 trial registry, and the Living Overview of Evidence platform were searched until January 29, 2021. Study Selection: The RCTs selected compared any type of convalescent plasma vs placebo or standard of care for patients with confirmed or suspected COVID-19 in any treatment setting. Data Extraction and Synthesis: Two reviewers independently extracted data on relevant clinical outcomes, trial characteristics, and patient characteristics and used the Cochrane Risk of Bias Assessment Tool. The primary analysis included peer-reviewed publications of RCTs only, whereas the secondary analysis included all publicly available RCT data (peer-reviewed publications, preprints, and press releases). Inverse variance-weighted meta-analyses were conducted to summarize the treatment effects. The certainty of the evidence was assessed using the Grading of Recommendations Assessment, Development, and Evaluation. Main Outcomes and Measures: All-cause mortality, length of hospital stay, clinical improvement, clinical deterioration, mechanical ventilation use, and serious adverse events. Results: A total of 1060 patients from 4 peer-reviewed RCTs and 10 722 patients from 6 other publicly available RCTs were included. The summary risk ratio (RR) for all-cause mortality with convalescent plasma in the 4 peer-reviewed RCTs was 0.93 (95% CI, 0.63 to 1.38), the absolute risk difference was -1.21% (95% CI, -5.29% to 2.88%), and there was low certainty of the evidence due to imprecision. Across all 10 RCTs, the summary RR was 1.02 (95% CI, 0.92 to 1.12) and there was moderate certainty of the evidence due to inclusion of unpublished data. Among the peer-reviewed RCTs, the summary hazard ratio was 1.17 (95% CI, 0.07 to 20.34) for length of hospital stay, the summary RR was 0.76 (95% CI, 0.20 to 2.87) for mechanical ventilation use (the absolute risk difference for mechanical ventilation use was -2.56% [95% CI, -13.16% to 8.05%]), and there was low certainty of the evidence due to imprecision for both outcomes. Limited data on clinical improvement, clinical deterioration, and serious adverse events showed no significant differences. Conclusions and Relevance: Treatment with convalescent plasma compared with placebo or standard of care was not significantly associated with a decrease in all-cause mortality or with any benefit for other clinical outcomes. The certainty of the evidence was low to moderate for all-cause mortality and low for other outcomes.

Problems with Evidence Assessment in COVID-19 Health Policy Impact Evaluation: A systematic review of study design and evidence strength.

INTRODUCTION: Assessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment. This study systematically reviewed the strength of evidence in the published COVID-19 policy impact evaluation literature. METHODS: We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on November 26, 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation, assessing what impact evaluation method was used, graphical display of outcomes data, functional form for the outcomes, timing between policy and impact, concurrent changes to the outcomes, and an overall rating. RESULTS: After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. The majority (n=23/36) of studies in our sample examined the impact of stay-at-home requirements. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-section), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 1/27 studies passed all of the above checks, and 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes. DISCUSSION: The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigor to be actionable by policymakers. This was largely driven by the circumstances under which policies were passed making it difficult to attribute changes in COVID-19 outcomes to particular policies. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

The worldwide clinical trial research response to the COVID-19 pandemic - the first 100 days.

Background: Never before have clinical trials drawn as much public attention as those testing interventions for COVID-19. We aimed to describe the worldwide COVID-19 clinical research response and its evolution over the first 100 days of the pandemic. Methods: Descriptive analysis of planned, ongoing or completed trials by April 9, 2020 testing any intervention to treat or prevent COVID-19, systematically identified in trial registries, preprint servers, and literature databases. A survey was conducted of all trials to assess their recruitment status up to July 6, 2020. Results: Most of the 689 trials (overall target sample size 396,366) were small (median sample size 120; interquartile range [IQR] 60-300) but randomized (75.8%; n=522) and were often conducted in China (51.1%; n=352) or the USA (11%; n=76). 525 trials (76.2%) planned to include 155,571 hospitalized patients, and 25 (3.6%) planned to include 96,821 health-care workers. Treatments were evaluated in 607 trials (88.1%), frequently antivirals (n=144) or antimalarials (n=112); 78 trials (11.3%) focused on prevention, including 14 vaccine trials. No trial investigated social distancing. Interventions tested in 11 trials with >5,000 participants were also tested in 169 smaller trials (median sample size 273; IQR 90-700). Hydroxychloroquine alone was investigated in 110 trials. While 414 trials (60.0%) expected completion in 2020, only 35 trials (4.1%; 3,071 participants) were completed by July 6. Of 112 trials with detailed recruitment information, 55 had recruited <20% of the targeted sample; 27 between 20-50%; and 30 over 50% (median 14.8% [IQR 2.0-62.0%]). Conclusions: The size and speed of the COVID-19 clinical trials agenda is unprecedented. However, most trials were small investigating a small fraction of treatment options. The feasibility of this research agenda is questionable, and many trials may end in futility, wasting research resources. Much better coordination is needed to respond to global health threats.

Limitations of the UNAIDS 90-90-90 metrics: a simulation-based comparison of cross-sectional and longitudinal metrics for the HIV care continuum.

OBJECTIVES: The Joint United Nations Programme on HIV/AIDS (UNAIDS) 90-90-90 and other cross-sectional metrics can lead to potentially counterintuitive conclusions when used to evaluate health systems' performance. This study demonstrates how time and population dynamics impact UNAIDS 90-90-90 metrics in comparison with a longitudinal analogue. DESIGN: A simplified simulation representing a hypothetical population was used to estimate and compare inference from UNAIDS 90-90-90 metrics and longitudinal metrics based on Kaplan-Meier-estimated 2-year probability of transition between stages. METHODS: We simulated a large cohort over 15 years. Everyone started out at risk for HIV, and then transitioned through the HIV care continuum based on fixed daily probabilities of acquiring HIV, learning status, entering care, initiating antiretroviral therapy (ART), and becoming virally suppressed, or dying. We varied the probability of ART initiation over three five-year periods (low, high, and low). We repeated the simulation with an increased probability of death. RESULTS: The cross-sectional probability of being on ART among persons who were diagnosed responded relatively slowly to changes in the rate of ART initiation. Increases in ART initiation rates caused apparent declines in the cross-sectional probability of being virally suppressed among persons who had initiated ART, despite no changes in the rate of viral suppression. In some cases, higher mortality resulted in the cross-sectional metrics implying improved healthcare system performance. The longitudinal continuum was robust to these issues. CONCLUSION: The UNAIDS 90-90-90 care continuum may lead to incorrect inference when used to evaluate health systems performance. We recommend that evaluation of HIV care delivery include longitudinal care continuum metrics wherever possible.

Cost-effectiveness analysis of tapentadol immediate release for the treatment of acute pain.

BACKGROUND: Tapentadol immediate-release (IR) tablets are indicated for the treatment of moderate to severe acute pain. In clinical trials, tapentadol IR effectively reduced moderate to severe pain with improved tolerability compared with oxycodone IR at doses providing comparable analgesia. OBJECTIVE: This analysis compared the cost-effectiveness of tapentadol IR with doses of oxycodone IR providing comparable analgesia in the outpatient treatment of acute postsurgical and nonsurgical pain. The perspective was that of a US managed care health plan as third-party payer. METHODS: A Markov model was developed to simulate clinical-economic outcomes for tapentadol IR 100 mg compared with oxycodone IR 15 mg in the treatment of acute postsurgical pain (3 days) and for tapentadol IR 50 mg compared with oxycodone IR 10 mg in the treatment of acute nonsurgical pain (10 days). The model simulated changes in pain relief; occurrence of opioid-related adverse events (AEs); opioid switching, discontinuation, and dose change; and number of quality-adjusted life-days (QALDs). Data inputs for the model were obtained from clinical trials, claims databases, surveys, Medicare fee schedules, and other published sources. Only direct costs were included. Drug costs were based on the wholesale acquisition cost. Prescription copayments were set at $5 for oxycodone IR and $25 for tapentadol IR. All costs were in 2008 US dollars. Sensitivity analyses were conducted on key model parameters. RESULTS: The cost of pain medication per patient was higher for tapentadol IR than for oxycodone IR in both the surgical pain setting ($15.23 vs $9.57, respectively) and the nonsurgical pain setting ($57.17 vs $21.31). However, this cost difference was offset by reductions in pharmacy and medical costs associated with the treatment of AEs and opioid switching/discontinuation, resulting in a lower mean treatment cost per patient for tapentadol IR 100 mg compared with oxycodone IR 15 mg in the treatment of acute surgical pain ($52.90 vs $55.99) and for tapentadol IR 50 mg compared with oxycodone IR 10 mg in the treatment of acute nonsurgical pain ($139.48 vs $144.79). Tapentadol IR also was associated with a greater mean number of treatment days with ≥30% improvement in pain intensity without opioid-related AEs compared with oxycodone IR and a greater mean number of QALDs (surgical pain: 1.73 vs 1.68; nonsurgical pain: 6.03 vs 4.92). Because both doses of tapentadol IR were dominant (ie, lower treatment costs and greater effectiveness) relative to the corresponding doses of oxycodone IR providing com- parable analgesia, incremental cost-effectiveness ratios were not calculated. CONCLUSION: The results of this model suggest that at doses providing comparable analgesia, tapentadol IR is a cost-effective alternative to oxycodone IR for the treatment of acute surgical and nonsurgical pain.